Conventional consumer market digital cameras typically involve image processing circuitry as well as a separate memory integrated circuit. The memory integrated circuit may, for example, be a synchronous dynamic random access memory (SDRAM).
Images are captured by an image sensor. The image sensor outputs a stream of raw image information that is stored into the memory. In the case of video or a rapid sequence of still images, the flow of image information can be a fairly constant stream. At the same time that raw image information later in the stream is being stored into the memory, image information earlier in the stream is being read out of the memory for image processing by a digital image pipeline (DIP). The digital image pipeline may include several processing blocks. A first image processing block may be operating on a first part of the image stream, whereas a second image processing block is operating on a second part of the image stream. Each block may read image information from the memory, perform processing, and then write the resulting processed information back out to the memory. Several blocks of the digital image pipeline may therefore attempt to access the memory at the same time.
Not only can different blocks of the digital image pipeline require access to the memory, but there may be other blocks of functionality on the digital camera that require memory access as well. In one example, the digital camera includes a JPEG compression block. The JPEG compression block reads an image out of the memory, compresses the image, and then outputs a compressed version of the image and stores the compressed version back to the memory. The digital camera typically has a zoom engine that reads an image out of the memory, creates a smaller version of the image called a “thumbnail,” and then writes this thumbnail back to the memory. The digital camera typically has a display that the user can use to view the image about to be captured. The thumbnail may be read out from memory and may be supplied to the display for viewing. The digital camera may also have the ability to overlay an icon or text information over an image. An on-screen display (OSD) engine may read a background image out of the memory, superimpose the icon or text or other visual feature on top of the background image, and then write the composite image back out to memory for viewing on the camera's display. A camera may also include a video encoder/decoder such as, for example, an MPEG2 codec. Image information may be read out of memory and sent through the MPEG2 codec. The resulting MPEG2 stream output by the MPEG2 codec may then be returned to memory.
The memory integrated circuit typically has a single access port that is used to write information into the memory and that is used to read information out of the memory. Due to the multiple different entities that need to read image information out of the memory integrated circuit, and the need to write image information into the memory integrated circuit, the access port of the memory integrated circuit is often times a throughput bottleneck in the camera. There is a limited amount of data that can be moved across the access port per unit time. In order for the camera to operate properly, the total amount of data to be moved either into or out of the memory by each of the accessing entities must total to a number less than the maximum memory access bandwidth of the memory integrated circuit.
Not only must the total memory access bandwidth required by all the accessing entities be less than the available memory access bandwidth available over the long term, but each of the accessing entities must not be made to wait too long to access the memory. The amount of time it takes to access the memory is sometimes referred to latency. If an accessing entity is made to wait too long, then operation of that entity may fail or be slowed or halted, thereby decreasing overall throughput of the camera. Some accessing entities may not be able to accommodate as much latency as other entities. The flow of video image information from the image sensor into the memory is one such process that typically can only tolerate a low amount of latency. If raw video image information being output from the image sensor cannot be stored within a certain amount of time, then it may be overwritten thereby resulting in the loss of raw image information. Other accessing entities, in contrast, can generally wait to access memory as long as over the long term those entities receive their required amount of access to the memory.
In one conventional digital camera, the total memory access bandwidth and latency issues are handled using an arbiter. Each of the accessing entities has its own dedicated DMA engine or engines. Within each accessing entity there may be sub-entities that access memory. When one of the accessing entities needs to access the memory, its DMA engine makes a request to the arbiter. If there is only one DMA engine making a request, then the request is granted and the accessing entity gains access to the memory. If there are multiple DMA engines making simultaneous requests, then one of the DMA engines is selected based on a strict priority or round robin arbitration scheme. If, for example, the DMA engine that moves raw image data from the image sensor into the memory is making a request at the same time that the zoom engine's DMA engine is making a request, then the DMA engine for the raw image data will typically have its request granted and the zoom engine's DMA engine will typically have to wait. To prevent latency problems, the system is designed so that so many high priority requests cannot be submitted in such a short period of time that the latency and bandwidth requirements of the lowest priority DMA engines are violated. Controlling when DMA engines can make requests and making sure that latency and throughput requirements of each of the requesting entities are not violated can give rise to difficult system design issues. If the system is changed, for example, then timing of the various DMA engines and the timing of their requests can change in complex ways. Reanalysis of the interplay between the various processing blocks of the system may be required. Flow shaping and scheduling are employed in the design of routers and switches in the networking and the telecommunications arts to handle bandwidth and latency issues. These techniques are, however, generally complex and tend to be expensive and cumbersome to implement. An inexpensive solution is desired that is suitable for use in a price-sensitive consumer market digital camera.